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METHODS AND STRUCTURES FOR AN 
EXTENSIBLE RAID STORAGE ARCHITECTURE 

Background of the invention 

1. Field of the Invention 

The invention relates to storage subsystem architectures and in particular 
to a RAID storage subsystem architecture that applies SAN principles and 
technology to the internal architecture of the storage subsystem. 
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2. Discussion of Related Art 

mputing storage subsystems are evolving at a rapid pace to require, at 
once, hig^Kpapacity, high performance and high reliability. Disk drive technology 
has evolveosto enable large capacities in individual disk drives. As applied in 
storage subsystems with multiple drives to achieve higher total storage capacity, 
each high capacity disk drive gives rise to performance bottlenecks as well as 
significant reliability^roblems. Where for example an entire request to store or 
retrieve data is direct^ to a single disk drive, the throughput of the storage 
system will be that of the\ingle disk drive and the reliability of the subsystem will 
be th^of a particular disk o^^ve. 

Reduradant arrays of inexpensive disks ("RAID") storage systems have 
Pressed the^ needs by providing redundancy for reliability and management 
techniques to acnieve higher performance. Specifically, RAID subsystems apply 
various managementstechniques (often referred to as RAID "levels") to provide 
redundancy in the stora^of data on the disk drives such that failure of a single 
disk drive does render theWire subsystem unusable. Other RAID techniques 
("striping") distribute the data oVer multiple disk drives to achieve the benefit of 
multiple disk drives processing a single larger I/O request to read or write data. 
Where N disk drives are used to process a single I/O request, the time to 
complete the request as compared to a single drive is on the order of 1/A/. 

The 'array" of multiple disk drives in a RAID storage subsystem is 
naged by a RAID storage controller device. The storage controller typically 
includes a general purpose microprocessor with associated program memory. 
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cache memoo^ for caching data serit to and from the disk drive array, "back-end" 
interfaces to afelapt the controller to W disk drive array (i.e., SCSI and/or Fibre 
Channel interface^ntrollers), a "front-bqd" interface to couple the controller to 
one or more host systern^etc. The storage>SQntroller manages the disk array to 
5 make the array appear to a nb§t computer as a l^fge single disk drive that offers 

improved performance and reliability as compared thab^f a single disk drive. 
£^i,C^r further enhance reliability and performance, RIAD subsystems also are 
^Ikfiown to utilizehqultiple such storage controllers. The multiple storage controller 
are often configurea\and managed to provide redundancy such that failure of a 

10 single storage controlW does not render the subsystem inaccessible. The 
multiple controllers may also be configured to enhance performance of the 
storage subsystem by providlKw parallel processing by multiple controllers of 
multiple host system I/O requestKThe load of I/O requests may therefore be 
distributed over the plurality of storage^ontrollers to reduce the total processing 

15 time required for a series of I/O requests it^t may be processed in parallel. 

Such multiple controller architectures still suffer from certain performance 
bottlenecks. For example, it is common that the multiple controllers share a 
common connection to the disk drives in the disk array. Shared use of the 
common disk interface can therefore become a performance restriction for 

20 multiple controllers in processing multiple I/O requests in parallel. Similarly, the 
number of I/O connections ("channels") for connecting the multiple controllers to 
host systems may be a bottleneck. 

Addition of disk drives without corresponding addition of communication 
channels and associated back-end control functionality could easily saturate 

25 existing disk channels. However, presently known architectures do not readily 
lend themselves to addition of disk drive communication channels independent of 
controllers having integrated front-end and back-end control functions. Present 
architectures generally require that the maximum anticipated bandwidth 
requirements of the back-end communication channels be anticipated in the 

30 original design and architecture of the storage subsystem. When applied to 
lower-end applications requiring only a portion of such capacity, the subsystem is 



2 



LSI Docket No.: 98-063 




"over designed" in that excess bandwidth capacity is unused and therefore 
wasted and costly. 

Some prior architectures called for "N-way" connectivity among the 
controllers and the disk drives. In other words, any number "N" of controllers 
5 shared access to a common set of disk drives via a common, single 
communication channel. However, such architectures can rapidly saturate the 
single, shared communication channel when additional disk drives are added to 
increase storage capacity. Even where multiple communication channels are 
utilized, the architecture calls for each controller to access each disk drive adding 
1 0 cost and complexity to each of the N controllers. 

In general, present high performance RAID storage subsystems suffer 
from lack of flexibility in configuring the multiple controllers and multiple disk 
storage devices or modules. It is therefore desirable to improve the flexibility of 
such configurations to permit easier enhancement of performance and reliability 
1 5 characteristics of a storage subsystem. 



Summary of the Invention 

£yefi/^'^y^^\ present invention solves the above and other problems, thereby 
advancing the state of the useful arts, by providing a storage subsystem 

20 architecture th^^ divides the controller function between front-end controller and 
back-end controW and that applies storage area network ("SAN") techniques 
and devices withirK the storage subsystem to interconnect the front-end 
controllers and back-ehd controllers. SAN components are known and applied 
outside the storage subsystem for interconnection of such storage subsystems to 

25 host computers and other cohnputing subsystems. In the context of this invention, 
SAN switches are applied vilmn the storage subsystem to permit more flexible 
configuration of front-end and B)ack-end control devices within the storage 
subsystem. 

A plurality of back-end storage controllers and a plurality of front-end 
30 controllers are configured within a storage subsystem interconnected by a SAN 
switching network that permits broad flexibility in interconnecting the various 
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controllers. The front-end controllers ("FECs") are dedicated to "front-end" 
interfacing to host computer systems and are devoid of circuits and functions to 
control the disk array devices. The back-end controllers ("BECs") are dedicated 
to "back-end" control of the disk arrays and are devoid of circuits and functions to 
interface directly with the attached host systems. In this architecture, the FECs 
and BECs are simpler than prior integral controllers that provided both front-end 
and back-end control functions. 

^^^^/E^h FEC and BEC includes a SAN interface to connect to the SAN 
^^/^Witches. The SAN switches therefore provide flexible interconnection between 
virtually anyViumber of front-end controller and any number of back-end 
controllers. Such a storage subsystem may thereby be flexibly configured to add 
additional back-enhxontrol where required for back-end performance or reliability 
enhancement and mav be configured to add additional front-end controller when 
required for front-end pehprmance and reliability. 

By providing such configuration flexibility and simpler FEC and BEC 
devices that segregate their respective functions, the storage subsystem is more 
scalable than prior known architectures. Additional FECs may be added to 
alleviate host communication bottlenecks independent of BEC control functions. 
Conversely, BECs may be added to alleviate disk communication bottlenecks 
independent of FEC control functions. 

Brief Description of the Drawings 

Figure 1 is a block diagram of a RAID storage subsystem as presently 
known in the art. 

Figure 2 is a block diagram of an exemplary RAID storage subsystem In 
accordance with the present invention. 

Figure 3 is a block diagram of a front-end controller of figure 2. 
Figure 4 is a block diagram of a back-end controller of figure 2. 



Detailed Description of the Preferred Embodiments 
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While the present invention is susceptible to various modifications and 
alternative forms, specific embodiments thereof have been shown by way of 
example in the drawings and will herein be described in detail. It should be 
understood, however, that it is not intended to limit the invention to the particular 
5 form disclosed, but on the contrary, the invention is to cover all modifications, 
equivalents, and alternatives falling within the spirit and scope of the invention as 
defined by the appended claims. 



Figure 1 is a block diagram of a typical multi-controller RAID storage 
10 subsystem 1 as presently practiced in the art. A plurality of storage controllers 100 
and 110 (Redundant Dual Access Controllers ("RDACs") #1 and #2) within the 
subsystem provide both front-end interfacing to hosts 170.. 174 via medium 160 and 
back-end interfacing to a pair of storage modules 120 and 130 via medium 150. 
The storage modules 120 and 130 each include a plurality of disk drives 122 and 
15 132, respectively. Each storage controller 100 and 110 is coupled to medium 160 
via a front-end interface element 102 and 112, respectively. Storage controller 120 
is coupled to both storage modules 120 and 130 via back-end interfaces 104 and 
106, respectively. Storage controller 110 is coupled to both storage modules 120 
and 130 via back-end interfaces 114 and 116, respectively and through 
20 communication media 1 50. 



^^JJ^ ^^Us known in the art, the host communication media 160 may be any of 
;efal weHrknown media including: parallel SCSI, Fibre Channel, Ethernet (or 
other local are^etwork media), etc. Similarly, it is known in the art that the back- 
end communicatiorhmedia 150 may be any of several well-known media including 
25 parallel SCSI, Fibre ChaJ^I, ATA, EIDE, etc. Those skilled in the art will recognize 
that depending upon the OTGtice of media elements 150 and 160 may include 
appropriate switches, hubs anck other connectivity devices as required for the 
particular communication medium. 
^J^'*^ i^his exemplary known\architecture provides redundant connectivity within 
the^torage subsystem between the storage controllers and the storage modules. 
As noted above, this known architecture is inflexible in terms of scalability in that 
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the front-end ds^ntrol functions (i.e., performed within 102) are integrated on a single 
controller along wjth the back-end control functions (i.e., performed within 104 and 
106). If the subsystem has a need for enhancing back-end perfomnance, additional 
back-end performance ihvWie fomi of back-end interface elements and functions are 
5 coupled with front-end controS^cuits and functions. Likewise, if additional front-end 
processing power for host gen^rajed I/O requests is required, the additional 
controllers are integrated with, potentiaHv extraneous back-end control circuits and 
functions. Furthermore, the interconnectiohsof additional controllers with existing 
storage modules may be cumbersome dependin^n the type of connections used. 
10 3*v^^\^re specifically, the front-end controllers perform processing related to 
''If t/ansactions^ with attached host computer systems and higher level storage 
Cii managementVunctions while back-end controllers perform processing related to 
%4 RAID management of the storage devices and lower level controls within the 
J:j storage subsysteirhyEach controller therefore addresses different aspects of the 
Q 15 overall perfomriancexyf the storage subsystem. Both front-end and back-end 
h controllers confront problems with available capacity to handle host I/O 
f^l transactions. The size anosfrequency of host I/O requests impacts the performance 
Q requirements of both the fron^nd controllers and the back-end controllers. Back- 
end controllers confront probleh;(s relating to interfacing with disk drives and the 
20 associated communication channels used therefore. In particular, the back-end 
controller is matched to a communibation channel bandwidth associated with a 
number disk drives. The configuration of back-end controllers is therefore 
preferably matched to the performance characteristics of the disk drive attached to 
it and the associated communication channeKbandwidth. A few high performance 
25 disk drives can saturate the communication clwinels used to communicate with 
back-end controllers. Additional communication bandwidth for disk drives may 
therefore require additional back-end controllers tOv accommodate the potential 
saturation of the disk interface channel. The neetis to scale the front-end 
transaction processing performance and high end storage management is largely 
30 distinct from the needs to scale the back-end performancevfor RAID management 
and lower level storage management functions. Though not enabled by prior 
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techniques, it isojseful to isolate these functions to permit independent scaling of 
the performance o^front-end control functions and scaling of the back-end control 
functions. 

Figure 2 is a block diagram showing the architecture of a storage system 2 

5 in accordance with the present invention wherein the front-end control circuits and 
functions are separated from the back-end control circuits and functions. As used 
herein, "front-end" refers principally to the host system interfacing functions. 
Exemplary of the functions performed by such front-end controllers are higher level 
I/O request processing such as RAID storage management for redundancy, RAID 

10 logical to physical storage mapping, hierarchical storage management, network file 
protocol support, high level data striping, backup and restore, routing of I/O 
requests among controllers, and management functions to map storage to data 
applications. As used herein, "back-end" refers to lower level control functions 
relating to disk drive interfacing and associated physical I/O operations on the disk 

1 5 drives. Exemplary of such back-end control functions are high availability storage 
functions (i.e., RAID management), high performance disk interfacing, high 
bandwidth I/O management, local device management and data management 
primitives such as data snapshots and data migration. 

Caching of data may occur in both front-end and back-end controllers - 

20 typically for different purposes and for enhancing performance of different aspects 
of the storage subsystem. Those skilled in the art will recognize that the definitions 
herein of high level or front-end functions as compared to lower level of back-end 
functions are matters of design choice. Other definitions and divisions of functions 
among the controllers are possible and within the scope of the present invention. 

25 Key to the invention is some division of functions between a front-end controller and 
a back-end controller allowing independent scaling of the controllers. 

The two layers (front-end and back-end) communicate via a SAN 
architecture layer preferably using an intelligent, structured interface protocol. The 
interface protocol may utilize a custom design protocol because this architecture is 

30 internal to the storage subsystem and need not be exposed external to the 
subsystem. In the alternative, the structured interface protocol may apply industry 
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Standards such as 1^0 or the Intel Virtual Interface Architecture. Again, such 
interface protocols and structures issues constitute well known design choices for 
those skilled in the art. 

In particular, storage system 2 includes a plurality of front-end control 

5 elements 200, 208 and 216. Each front-end controller includes one or more front- 
end interface elements (202, 210 and 218, respectively) to connect the front-end 
control element (FEC) to one or more host systems 170.. 174 via a host 
communication media 160.. 161. FECs may be connected to a plurality of host 
system communication media as required for flexible connectivity to host systems. 

10 For example, media 160 and 161 may be separate segments of a common 
communication media type or may even be different types. As noted above, the 
communication medium used between FECs and host systems may be any of 
several well-known types as discussed above. 

Each FEC (200, 208, 216) also includes one or more intra-subsystem SAN 

15 interfaces (204 and 206, 212 and 214, and 220 and 222, respectively). Intra- 
subsystem SAN interfaces 204, 206, 212, 214, 220 and 222 are referred to as 
"intra-subsystem" to distinguish from SAN interfaces that may be used In a storage 
subsystem to connect to SAN components external to the storage subsystem. Such 
external SAN interfaces are not relevant to the operation and structure of the 

20 present invention. As used herein below "SAN interface" refers to intra-subsystem 
SAN interfaces as distinct from any SAN interfaces that may be present on a 
controller for interfacing external to the storage subsystem. 

Each FEC includes one or more SAN interface elements connecting the 
FEC to the SAN switches 250 and 252 via SAN communication media 254. There 

25 are preferably at least two SAN switches 250 and 252 to permit such redundant 
connectivity from the front-end control elements to the plurality of back-end control 
elements discussed below. There may be any number of such redundant links but 
in the preferred embodiment, two links from each front-end control element, one to 
each of two SAN switches, is considered necessary and sufficient. Where reliability 

30 of the front-end control communication with the back-end control elements is 
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deemed less important, a single connection between a front-end control element 

and the SAN switches may be adequate. 

Storage system 200 also includes a plurality of back-end control elements 

260, 264, 268 and 272 preferably configured as shown in redundant pairs (260 and 
5 264 as a first pair and 268 and 272 as a second pair). Each back-end control 

element (BEC) includes a SAN interface element (262, 266, 270 and 274, 

respectively). Each BEC of a redundant pair is connected to one of the two 

redundant SAN switches 250 and 252 via SAN communication media 256. 

Specifically as exemplified in figure 2, back-end control element 260 (BEC) 
10 connects to SAN switch 250 via SAN interface 262. BEC 264 connects to SAN 

switch 252 via SAN interface 266. BEC 268 connects to SAN switch 252 via SAN 

interface 270 and lastly, BEC 272 connects to SAN switch 250 via SAN interface 

274. 

Each BEC connects to a storage module 280 or 290 comprised of a 

15 plurality of disk drives 282 and 292, respectively. Each BEC of a redundant pair 
preferably connects to one of the storage modules. For example, as shown in 
figure 2, BEC 260 connects to storage module 280 via media 150 and BEC 264, 
the other BEC of the redundant pair of 260 and 264, also connects to storage 
module 280 via media 150. It is also possible for each BEC to provide a pair of 

20 redundant links to its associated storage module. For example, as shown in 
figure 2, redundant pair of BECs 268 and 272 are each redundantly connected to 
storage module 290 via a redundant pair of communication links in media 150. 
As noted above, communication media 150 between the BECs and the storage 
modules may be any over several well-known types as discussed above. 

25 Those skilled in the art will recognize that the specific configuration 

(topology) shown in figure 2 is intended merely as exemplary of one possible 
embodiment in accordance with the present invention. The present invention 
provides for the segregation of front-end control functions and back-end control 
functions into distinct circuits with a SAN architecture interconnecting the 

30 elements. A wide variety of alternate configurations and topologies will be 
recognized by those skilled in the art. Further, the number of FECs, BECs and 
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SAN switches and the grouping of those devices into pairs, is intended merely as 
exemplary of one preferred embodiment. Any number of FECs, BECs and SAN 
switches may be configured in a system in accordance with the present 
invention. In a particular application, the number of such controllers and SAN 
5 switches is determined by matching the bandwidth and transaction processing 
capability of the components with the subsystem requirements for that 
application. The individual modules and components (FECs, BECs, disk drives, 
SAN switches, etc.) of the storage subsystem in accordance with the present 
invention may be dynamically reconfigured by a user to modify performance 
1 0 characteristics to fit changing demands on the storage subsystem. 

The present invention expresses the preference for at least pairs of SAN 
01 switches and pairs of BECs to ensure redundancy throughout the connections 
H from a host system through to the individual disk drive devices. Any number of 
y FECs, BECs and SAN switches, paired or not, may be configured within the 
Q 15 intended scope of the present inventions. 

p. As noted above, the SAN switches (250 and 252) and associated SAN 

Cn communication media 254 and 256 may apply any of several existing SAN 
Q architectures. The SAN switches and associated communication media may be 
U any that allows the passing of data and I/O requests between the FECs and the 
20 BECs with low latency (i.e., less than 10 microseconds). Typical of such 
devices/media are PCI buses, local area network (LAN) connections (i.e., 
Ethernet or Gigabit Ethernet, etc.). Fibre Channel SAN switch devices and 
media, InfiniBand (www.infinibandta.org) and ServerNet (developed by Tandem 
and presently sold by Compaq). The ideal configuration involves a switch that 
25 allows for bandwidths that scale with the number of devices (FECs and BECs) 
that are added to the SAN. Present market forces and technology factors 
suggest that InfiniBand is a preferred embodiment of the SAN communication 
media. 

Figure 3 is a block diagram of a typical FEC device in accordance with the 
30 present invention. As noted above, the FEC of the present invention is similar to 
a storage controller as presently known in the art and as shown in figure 1 except 
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that it is devoid of back-end control functions and circuits. Rather, the FEC has a 
redundant SAN interface to permit flexible connectivity to back-end control 
elements through the SAN layer. 

In particular, FEC 200 is shown in additional detail in figure 2. FEC 200 is 

5 intended as exemplary of all FECs shown in figure 2 above. In the preferred 
embodiment, FECs are not identical devices. As noted above, each FEC may 
provide a different type of host (front-end) interface to permit added flexibility in 
the connectivity of the storage system. The different types of host interfaces may 
include different physical interfaces and protocols or merely different logical 

10 interfaces provided on a common physical medium. In addition, different front- 
end interfaces may provide varying functions for particular connection 
application. For example, network file protocols may be directly supported in a 
particular FEC while another FEC may provide only lower level block level 
access interface functions. 

15 In the preferred embodiment. FEC 200 includes a general purpose CPU 

300 that controls overall operation of the FEC and processes I/O requests 
received from the front-end interface element 202 and received from the back- 
end devices connected through SAN interfaces 204 and 206. Programmed 
instructions and data for operation of CPU 300 are stored in program memory 

20 304. Data sent to or from the host systems or the BECs is cached in cache 
memory 302 to improve controller performance. DMA 306 assists CPU 300 in 
transferring data among the various components. All components communicate 
via processor bus 350. 

Those skilled in the art will recognize that the block diagram of figure 3 is 

25 intended merely as exemplary or suggestive of the design of an FEC. The 
specific compliment of components associated with the CPU and the specific bus 
or buses that interconnect those components is a matter of design choice well- 
known to those skilled in the art. For example, a front-end controller may 
optionally include RAID parity assist (RPA) computation devices for other higher 

30 level RAID management support in the FEC. Key to the FEC design shown in 
figure 3 is that the FEC is devoid of back-end disk drive interface components. 
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Rather, that function is segregated onto a back-end controller element. The FEC 
therefore preferably performs necessary mapping of logical storage addresses 
(supplied by host I/O requests) into physical storage locations conveyed to 
appropriate BECs to perform the host requested I/O operation. The SAN 

5 interfaces 204 and 206 permit flexible interconnection of the FEC with a number 
of BEC elements via the SAN intermediate layer. Further, as noted above, 
intelligent I/O interfacing protocols and APIs are preferably implemented within 
the FEC and BEC to permit a structured, standardized interface between the 
layers through the SAN switch intermediate communication layer. 

10 Figure 4 is a block diagram of an exemplary back-end control element 260 

(BEC) in accordance with the present invention. BEC 260 is representative of all 
BEC elements shown above in figure 2. BECs are preferably identical in design, 
though as noted they may vary in accordance with specific needs of a particular 
storage system application. For example, different BECs within a storage system 

15 may each provide a different back-end interface medium to connect to a set of 
disk drives. A first BEC in a storage system may use parallel SCSI for example to 
connect to a storage module while a second BEC in the same storage system 
may use Fibre Channel to connect to a storage module. Similarly, a first BEC 
may provide a single connection to a storage module while another BEC in the 

20 same system may provide redundant links to another or the same storage 
module. Or, for example, groups of BECs may be tuned for different performance 
characteristics. Some BECs may be tuned to high bandwidth back-end 
performance requirements while others could be tuned high I/O transaction rate 
requirements. Still other BECs may be tuned for tape storage as distinct from 

25 disk storage. Such flexible configurations are useful in hierarchical storage 
management applications where a multitude of storage media are incorporated 
into a single storage subsystem each medium having different access and 
performance characteristics. 

As above, the BEC performs the lower level functions of interfacing with 

30 the disk drives directly. Lower level physical I/O operations are performed by the 
BEC. As noted above, a key to the architecture of the BEC of the present 
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invention Is that it is devoid of functions and associated circuits for performing 
host interfacing (front-end interfacing). Othenwise, BECs are substantially similar 
to the general structure of FECs. Programmed instructions and data for operation 
of CPU 300 are stored in program memory 304. Data sent to or from the disk 
5 drives or the FECs is cached in cache memory 302 to improve controller 
performance. DMA 306 assists CPU 300 in transferring data among the various 
components. As noted above, RAID storage management functions are 
preferably performed within BEC 260. RPA 308 (RAID Parity Assist) aids CPU 
300 in rapidly computing parity values for RAID storage management functions 
10 within the BEC. All components communicate via processor bus 450. 

^ particular, BEC 260 includes one or more SAN Interfaces 262 to 
:6nnect\o the SAN communication media 256. The SAN interfaces 262 are 
Q coupled Vila bus 450 to disk interfaces 400 and 402 which, in turn, coupled via 
ill bus 150 to Storage modules and/or individual disk drives. As shown in figure 4, 
^ 15 disk interfaced 400 and 402 include all intelligence required to Interface with a 

f!2 \ 

front-end controNelement via bus 450 and SAN interface 262. Those skilled in the 
Jj{ art will recognize that in particular applications it may be beneficial to implement 
W the FEC and BEC as identical hardware components each implementing its 
r? particular designated fuhction. Such identity of the hardware components permits 
-==^ 20 more flexible replacement^f spare parts in the subsystem. Further, those skilled 
in the art will recognize tharmany of the components In an FEC or BEC may be 
integrated into higher level n^egrated circuits incorporating many discrete 
functions into a VLSI custom cira^it. Such design choices are well-known to 
those skilled in the art. Key to the BECxrf the present invention is that it is devoid 
25 of front-end functions and associated circLHt^ Rather, it performs only the back- 
end functions of low level disk drive command processing. Interfacing with higher 
level front-end control elements is provided via the^SAN interfaces of the BEC. 

The present invention as described above provides a key benefit in that 
the architecture can be flexibly scaled to different bandwidth requirements unique 
30 to particular applications. As back-end performance becomes a bottleneck, 
additional BECs may be easily integrated. Likewise, as front-end I/O processing 
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performance becomes a bottleneck in system throughput, additional FECs may 
be added to improve I/O processing performance. Further, as new or additional 
host interface channels or protocols are required, additional FECs having 
different host channel interfaces and/or protocols may be added. The 
5 architecture of the present invention therefore improves flexibility and scalability 
of the storage subsystem to allow customization and adaptation to particular 
needs of particular applications. 

While the invention has been illustrated and described in detail in the 
10 drawings and foregoing description, such illustration and description is to be 
considered as exemplary and not restrictive in character, it being understood that 
only the preferred embodiment and minor variants thereof have been shown and 
described and that all changes and modifications that come within the spirit of the 
invention are desired to be protected. 
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